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Introduction to the Study 


Through its program review process, the Illinois Community College Board (ICCB) coordinates a statewide 
system of review of programs of study provided by community colleges in Illinois. The intent of this review 
process is to support colleges in making campus-level planning and decision-making related to the quality, cost 
effectiveness, assessment, and improvement of programs. In addition, the review process identifies programs 
that should be discontinued or scaled-back and offers best practices and other exemplary innovations that can be 
scaled internally or externally within the college. A significantly modified 2017-2021 program review manual 
and supporting materials are provided by the ICCB to guide the process. As such, evaluation of the new manual, 
as well as supportive processes and materials, presents a timely opportunity to provide formative feedback 
intended to improve the efficacy of the process. 


Goals 


OCCRL is conducting an evaluation of the ICCB program review process for career and technical education 
(CTE) programs. The first phase of the study, summarized in this report, focuses on the first cycle of the ICCB’s 
new program review process that ended September 1, 2017. OCCRL has four goals for this evaluation study: 


1) Improve the efficiency and efficacy of the program review process by identifying challenges, 
redundancies, and omissions, and provide recommendations for refining the process. 

2) Examine variations of the program review process across institutional contexts and institutional 
identities (e.g., rural/suburban/urban, minority-serving institutions/predominately white institutions, 
small/large student populations) to understand how the process is utilized across diverse institutions 
throughout Illinois. 

3) Identify professional development, technical support, and supplemental materials that could improve 
outcomes associated with the program review. 

4) Enhance the application of program review findings in colleges’ campus-level programmatic planning 
and decision-making. 


Framework 


OCCRL framed this study around the concept of program review as a tool for supporting student-centered, 
equity-minded, and evidence-driven change at Illinois community colleges. This course of action was based on 
ICCB’s desire to see a program review used to support decision-making at two levels: the campus level and the 
program level. In addition, ICCB included new questions in the program review manual that foster exploration 
of disaggregated data and identification and discussion of inequities at the program level, which will help frame 
the study. Supporting the effective use of evidence or an evidence-based culture can have direct effects on the 
quality and rigor of programs and the outcomes of the students who are engaged in those programs (Kerrigan 
& Jenkins, 2013; Marchant & Paulson, 2009). There are at least seven critical environmental and institutional 
factors that affect an institution’s ability to foster a culture of evidence and to support equity-minded use of 
that evidence (Arenth, Bennett, Bernadotte, Carnahan, Dube, Thompson, & Walton, 2017; Bragg, Bennett, 

& McCambly; Copland, Knapp, & Swinnerton, 2009; Copeland, Knapp, & Swinnerton, 2009; Spurlock & 
Johnston, 2012). These seven environmental and institutional factors include: 


e Leadership. A culture of evidence is championed throughout the college through informal and formal 
leadership. 

e Strategic investments. The college strategically invests resources in student-centered improvements 
directed toward programs and pathways. 

e Equity-guided. A commitment is shared throughout the college to systemic and localized changes to 


improve equity for underserved students. 

e Engagement. Diverse viewpoints and roles are encouraged and sought after and contribute to ongoing 
improvement processes. 

e Statewide guidance and governance. The state provides clear and supportive standards and policies 
that support both accountability and data-driven change. 

e Data literacy. Institutionally and individually, the college supports access to data and the data literacy 
necessary to frame inquiries and to effectively collect, operationalize, analyze, and interpret data. 

e Infrastructure and information technologies. Systems are available to support the collecting, storing, 
accessing, analyzing, and disseminating of data throughout the college. 


Further, in order to build data into evidence, institutions must balance a three-part evidence-use cycle: a) the 
production of evidence, b) sense making, and c) evidence use (Arenth et al., 2017; Copeland et al., 2009). 
Producing the evidence extends substantially beyond the collection of data. It also includes framing the inquiry, 
accessing the data, assessing the quality and use of the data, and analyzing the data. In the second stage— 
making sense of the data—data is translated into meaning through interpretation. This might be, in essence, 
where the data is translated into evidence and involves consideration of alternative perspectives, challenging 
initial assumptions, and identifying implications. The final stage—utilizing the evidence to effect change— 
includes formulating a plan to support the timely use of the evidence, communicating the findings, taking 
evidence-based action, and reflecting on learning and inquiry. 


Evaluation Questions 
To address these goals, OCCRL attempted to answer three evaluation questions: 


1) What institutional and environmental factors within the context of program review effect institutions’ 
culture of evidence, and how do these factors vary across institutions in the state? 

2) What are the substantive challenges that influence the design, implementation, and impact of the 
program review in Illinois, and how might these challenges be remedied? 

3) What are critical design considerations for advancing the future of program review processes at 
community colleges in Illinois? 


Methods 


This evaluation was a combination of a developmental and participatory approach. Developmental evaluation, a 
pragmatic utilization-focused approach, provides a framing for engaging with “emergent and dynamic realities 
in complex environments” (Patton, 2011, p. 1). There is a high level of variation among I/linois community 
colleges and the programs nested within them, which results in a high level of variation and complexity among 
the environmental and institutional factors that impact institutions’ culture of evidence use. 


The program review process is also an emerging and dynamic process, with ongoing changes at state, college, 
and program levels that involve a large number of stakeholders with different roles and involvements in the 
process. It is by nature part of a complex, adaptive system that is nonlinear, emergent, dynamic, adaptive, 
uncertain, and co-evolutionary (Patton, 2011, p. 8). To meet the challenge of evaluating such a system, we 
focused on working closely with individuals within the field who were currently engaged in the work and who 
had developed expertise working within the system, allowing them to suggest improvements and envision 

its future. We adopted an intentionally participatory approach by leaving our evaluation questions and focus- 
group protocols open enough to allow participants to lead the conversations in a direction that addressed their 
greatest priorities, challenges, and strategies for moving the program review forward. This too is in line with 


developmental evaluation. Instead of attempting to evaluate the process holistically, our evaluation captured and 
attempted to address what is most relevant to the participants of the study. 


Finally, the nature of a developmental evaluation is to be utilitarian and, in this case, formative, as we work to 
support the program review process in Illinois community colleges. And though the lessons captured in this 
report are not generalizable, we hope they are informative for those seeking to improve program reviews or 
similar processes in different contexts. 


The primary source of data for this evaluation was a series of three events that OCCRL hosted in central Illinois 
that drew 49 community college practitioners, including institutional researchers, chief information officers, 
administrators, and faculty from representative colleges throughout Illinois. At the first of these two events, 
OCCRL researchers facilitated a series of focus groups based on the studies framework. The first event, held in 
February 2018, focused on environmental and institutional factors that influence program review. In deference 
to the limited time as well as our desire to ensure adequate depth on each topic covered, we selected six factors 
to focus on. Data was collected on data literacy, engagement, equity-guided, infrastructure and information 
technologies, leadership, and statewide guidance and governance. 


The second event, held in April 2018, focused on the evidence-use cycle. During this event, focus-group 
sessions centered on colleges’ processes for producing evidence, making sense of that evidence, and utilizing 
the accumulated evidence. Preliminary analysis from the first two events was used to identify common, 
substantive challenges faced by colleges that impact the design, implementation, and impact of program review 
in Illinois. 


In June 2018, a third and final event was held. In contrast to the first two events, this one was a design lab in 
which participants worked in small groups on targeted challenges or tasks related to program reviews. Their 
design activities included: 


1) responding to a design challenge identified in the focus-group data from the first two events 

2) providing a critical review of the ICCB’s CTE Program Review Template 

3) identifying critical design considerations for advancing the future of program review at community 
colleges in Illinois. 


Just under 19.5 hours of audio files were captured from the focus groups and the design labs (only their 
summary presentations were recorded at the design-lab sessions). These recordings were transcribed for the 
purposes of analysis. The data were analyzed using the framework of the study, with a focus on identifying 

key concepts and themes that reflected the priorities expressed by the study participants, while allowing us to 
address goals and evaluation questions (Krueger & Casey, 2009). Agendas for the February, April, and June 
events are provided in Appendixes A-C. Protocols for the events are available from the corresponding author by 
request. 


Throughout this evaluation process, the OCCRL research team triangulated data from multiple sources in an 
effort to identify areas in which alignment or additional supports were necessary to improve engagement with 
the program review process and the use of program review findings. This additional data was also critical to the 
design of the evaluation itself and influenced the nature of our findings and recommendations. 


First, prior to the design of the project, the OCCRL team reviewed the new 2017-2021 program manual and 
supportive materials and did a review of the existent literature on data-driven decision-making, evidence-driven 
culture, equity-driven change, and program review. This allowed us to create the framework for the study and 
was influential in our attempts to ensure that our sample reflected substantial institutional variation to account 
for the variance we anticipated among the institutional and environmental factors that create the context for 


program review at institutions statewide. 


Second, OCCRL researchers conducted a document analysis of samples of the program reviews submitted by 
the CTE program in the September 1, 2017, cycle. The team looked for patterns in the data that indicated areas 
where the program review process may have been unclear, or where additional support was necessary for the 
review to be accurate and effective. Researchers observed inconsistences and other patterns within the reviews 
that helped inform the questions included in the focus groups’ protocols and the challenges included in the 
design lab. 


Participants 


Participants were recruited through an open call for participation that was distributed by the ICCB. The call for 
participation requested that colleges and staff with expertise in the program review processes in Illinois send 
two to three colleagues such as administrators, deans, faculty members, institutional research staff, student- 
support services, or other support staff. Institutions that were interested in participating were asked to send a 
letter to ICCB detailing their interest, level of expertise, dedication to the project, and group members. Thirteen 
colleges responded, and these letters were reviewed by OCCRL and ICCB. 


The teams attempted to understand to what extent the sample of colleges that had responded reflected the 
college system in Illinois. Factors included the size of the institution, the demographics of the students served, 
the levels of resources available to institutions, and urbanity of the institution. Based on both groups assessment, 
ICCB did a targeted recruitment of representatives from two additional districts in the state, ensuring a more 
representative sample of participating institutions. Following the formal call for participants, an additional six 
colleges volunteered to participate in the study. Overall, there were 49 participants from 21 colleges, with 29 of 
the participants attending all three-research events. 


The participants in the study brought with them substantial depth of expertise in program review, having 
engaged in it directly for an extended period. For most participants, this meant they were engaged directly in the 
work for at least five years. However, many of them had program review experience in excess of 10 years, with 
a couple of participants having more than 20 to 25 years of experience. 


Additionally, participants were all recognized by their institutions for their internal contributions to the program 
review process, with most having substantially and meaningfully contributed to the evolution and redesign 

of the program review processes at their institutions. Additionally, for the majority of the participants, this 

also included actively engaging, through either coaching or mentoring, either formally or informally, others at 
their institution in the program review process. Often this occurred through a cyclic and ongoing process that 
involved providing intensive amounts of professional development. 


Lastly, the participants had a high level of familiarity and comfort with institutional data and processes. Most of 
them held administrative or staff titles, which included seven vice presidents, 18 deans, and 22 individuals who 
were directors or managers. Among the participants there were only two full-time faculty members, though 
some of the individuals carried a teaching load. In addition, 40% of the participants worked in roles in which 
evidence use and data culture were central facets of their work. Specifically, 20 of the individuals held jobs in 
institutional research or assessment. A couple of the participants were active scholars who were contributing 

to program review practices beyond their institution. This included mentoring peers at other institutions, 
conducting webinars, presenting at conferences, and publishing on the topic of program review. 


Environmental and Institutional Factors 
Data Literacy 


Data literacy is something that is core to program review and is, frankly, important to every decision we 
make. 


I would argue that your data is your reality and that sometimes your perception and your reality just do 
not line up, but you can t argue with the data. 


Data literacy is hard, and working with data can be really frustrating. It takes a lot of work to get people 
comfortable. 


The aforementioned statements expressed by focus group participants demonstrate the value they place on 
data-informed, institutional decision-making on their campuses. The purpose of these focus groups was to gain 
a more robust understanding of the role data played in the everyday operations at their respective colleges, 
including the program review process, as well as to glean insight into processes and roles associated with data 
gathering, analysis, and dissemination. 


As noted above, participants felt strongly that data (both quantitative and qualitative) should be used to inform 
decision-making. Although it was noted that standardization of measures was key to the accuracy and validity of 
findings, as well as resultant policy decisions, participants related that input from diverse voices is critical when 
the state is considering setting such definitions. Standardized operationalization of variables across colleges was 
seen as problematic to many participants. One participant who commented on the standardization process that 
her college has engaged in said: 


It used to be that we could have retention in one area compared to someone else who calculated their 
retention in another area, and they used apple as one of the instances and berries in another instance. 


In addition to reaching agreed-upon operationalization of data points, colleges must activate engagement and 
mentoring skills to make comfortable those who are initially intimidated by the idea of data. Varied degrees of 
data literacy and proficiency in analysis techniques complicate the program review process. As one participant 
said: 


When we talk about data literacy, I keep thinking of it as this proficiency spectrum where some folks are 
really comfortable swimming in the deep end and thinking about what it means and figuring out what 
pieces really mean to us. 


The focus groups continued with participants completing a Data Capacity Survey to measure their comfort 
level and proficiency in utilizing data on their campuses. Results from this self-reported measure showed that 
participants were most comfortable with analyzing and interpreting findings (4.07 and 4.06 respectively). These 
findings are not surprising given the composition of the focus-group participants, which was professionals who 
had extensive experience with program review. The survey reveals that proficiency with data varies across 
institutions; most drastically in the area of collecting data (ranges between 1.75-5.00, see Table 1). 


Table 1 
Results of Data-capacity Survey. Measured Self-Reported Skill Proficiency in Four Areas 
(5 as Highest Score Possible) 


Area Average High Low 
Accessing Existing Data 3.70 (intermediate) 5 Ds 
Collecting Data 3.88 (intermediate) 5 ES 
Analyzing Data 4.07 (advanced) 5 25) 
Interpreting Findings 4.06 (advanced) 5 DSi) 


Institutional researchers (IR) and institutional research offices play a vital role in collection, interpretation, and 
dissemination of program and institutional data. During program review, IR is especially important. As one 
participant noted, “We couldn t do it (program review) without them.” 


On the point of standardization, many participants credited IR with having the expertise and capacity to 
operationalize data points and disseminate these definitions across college programs. As one participant said, 
“IR is the shepherds keeper of those data definitions...” IR was additionally credited by the participants as an 
entity that accurately highlights limitations and challenges brought forth from the data. 


Furthermore, many institutions reported that IR is responsible for creating and maintaining team collaboration 
sites such as TaskStream and Sharepoint. In the absence of formal institutional researchers, or an institutional 
research office, participants reported that professionals within Information Technology (IT) absorbed functions 
normally related to institutional research. 


From a process perspective, focus-group participants detailed the critical role of IR in program review, with IR 
playing a key leadership role in program review at most institutions. Many spoke of institutional researchers 
who drafted templates for each program at the college to aide in the uniformity of reporting, and to add clarity 
to what was expected from reporting agencies such as ICCB and the Higher Learning Commission (HLC), as 
well as to prepare for annual and bi-annual internal program reviews. An IR representative spoke to this process 
by stating: 


We are providing some kind of guidance on how exactly to interpret the data, so we prepare the 
templates, and each template is supported by a data set to look at and use in answering those questions. 


A distinct, and possibly underreported, function of IR is classroom-level assessment. Though the crux of IR 
involves measuring program outcomes on an institutional level, at least two participants discussed ways IR 
supports faculty in assessing student learning outcomes and effectiveness of classroom activities in meeting 
certain standards: 


I'd say maybe 25 to 30 percent of what we do is consulting. Faculty have come to me and said, “I want 
to research ‘this’ with my students in my classes. What do you suggest for my methodology? What are 
some ideas?” And I get to learn more about their classes, too. 


A different IR representative echoed this function of IR: 
We offer support in that way (student assessment) if departments want help designing things, collecting 


the data, analyzing the data. We are at different places in that some departments take us up on that, and 
some departments do it on their own. 


As challenging as program review can be for colleges, the process itself can build and solidify bonds across 
college departments. Additionally, a successful program review demands that college wide stakeholders 
effectively communicate what their programs are doing and how they are doing it. One participant commented 
on a lesson learned: 


I think one of the most positive things I have gotten from being involved with program review is that 

you are having conversations with every person at the institution and understanding what their role 

is for students’ support and how that can be interpreted by different people.... As a result of those 

types of conversations, and our strategic planning initiative, we have built programmatic KPIs (key 
performance indicators) based on program review, and we use that in assessing programs for our quality 
initiative for HLC. 


Engagement 


One key institutional factor in equity-minded, evidence-driven change processes is engagement. Specifically, 
institutions in which diverse viewpoints and roles are engaged and contribute to ongoing improvement 
processes support a culture of evidence and student-centered, equity-guided, continuous improvement (Arenth, 
et al., 2017; Spurluck & Johnston, 2012). In this section, we set out to understand how Illinois community 
colleges engage internal and external stakeholders and promote commitment throughout the program review 
process. Furthermore, we briefly discussed mechanisms colleges use to activate diverse roles and incorporate 

a variety of viewpoints to foster collaboration and ongoing improvement processes. To address these factors, 
focus-group participants engaged in activities and dialogue that ultimately demonstrated the variance and 
overlap between how participating colleges cultivated engagement and commitment during program review. 


The Who Contributes activity asked participants to estimate the percentage of contributions provided during the 
process of program review within the following areas of their respective colleges: administration, faculty, and 
staff. 


Table 2 
Results from the Who Contributes Worksheet Documenting the Perceived Percentage of 
Contributions across College Sectors. 


Area of College Highest % Lowest % Average % Mode 
Administration 85% 10% 33% 25% 
Faculty 75% 15% 42% 40% 
Staff 45% 0% 24% 30% 


As shown in Table 2, participants estimated, on average, that faculty contributed the most to program review 
(42%) and staff contributed the least (24%). It must be noted that a few colleges (n = 4) stated that staff did not 
contribute at all to program review on their campuses. As the table indicates, the most frequent estimate of staff 
contribution was 30%. 


After completing the worksheet in small groups, members discussed responses with their peers. Participants 
reported utilizing a diverse array of strategies on their campuses to foster collaboration and engagement during 
program review. Some participants reported holding program review team meetings in the fall and/or spring 
semesters. These meetings included college wide representation in which team members discussed reporting 
templates, data sources, and the timelines of drafts and final products. These team meetings were examined as a 
way to clarify stakeholders’ questions and concerns. 


Participants discussed team sites as a popular organizing mechanism that fostered collaboration across campus 
and promoted transparency during program review. At least one college reported that its institution’s team site 
housed archived program reviews that served as a reference for future reporting needs. 


Group members commented on the importance of ensuring that all who are reviewing their programs have an 
understanding of where information being collected is going, and why it is being collected. One participant 
noted: 


I think that it is important that if they (student services professionals) are going to spend their time 
reviewing their program and not knowing where all that information is going, they are not invested in it. 
I would rather get them invested, as every five years we are engaged in this. 


In an attempt to bolster accuracy in reporting as well as trust, participants discussed the importance of 
cultivating an environment of understanding and nonjudgement. One participant disclosed that she informs 
faculty, program coordinators, and deans that they are not being negatively appraised on where their program 
currently is: 


I dont need you to scramble around and try to fix things that you are seeing that are problematic. Tell 
me what you are seeing and what you are going to do to address them.... They [[CCB] want us to own 
our warts and figure out what we are going to do.... In some ways that elevates some of the pressure, 
too. 


Finally, participants reported an apparent disconnect between faculty content expertise and their comfortability 
with assessment, as well as reporting their respective program learning outcomes and developing plans for 
addressing areas of concern. One participant shared an example of how her college addresses this issue: By 
mentoring faculty in how to respond to questions about strengthening their programs, faculty avoided justifying 
the continuation of their respective programs. Moreover, it was noted that faculty at the same college are 
encouraged to engage employers to better understand what skills and knowledge students need to become 
employable in that particular industry, as well as discuss labor projections and revamping curriculum, if 
necessary. 


Equity-Guided 


Given that there is an increased focus on student outcomes in CTE programming, it is important to ensure that 
institutions look for ways to guarantee equity across the board. We define “equity-guided” as a commitment that 
is shared throughout the college geared toward fostering systemic and localized changes that improve equity for 
underserved students. Participants were asked to share how program review supports student-centered change, 
and the efforts colleges make to improve educational equity in their programs and pathways. 


Inclusion of Equity 

Participants were asked to share their understanding of how equity fits within the program review process. 
Generally, participants demonstrated a positive reaction to seeing equity explicitly mentioned and explored 
within program review. “Nice to see equity and equity gaps explicitly articulated in the new template,” one 
person noted. In the session, participants were encouraged to share their thoughts on equity. Common phrases 


from participants included: “ownership over the review process”; “creating opportunities for integration and 
crossover”; and “equity as access.” 


Shifting from the positive aspects of equity to the opportunities that equity presents, several participants 
highlighted the ability to do more. One participant said, “Looking at equity is great, but the next step should 
be building in what you are going to do about it and being very specific about how you’re going to close those 
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gaps. Similarly, another participant said, “It’s great being aware of the equity issues, but how are we specifically 
addressing what’s causing the inequities.” Another participant addressed the need to look at outcomes and 
intake, asking, ““What is the population of your programs, where are students coming from, and is your school 
near the district?” Focusing on equity within program review from a broader view, another participant said, 
“When I think about it [equity] in program review, I focus on what do we have the ability to have control over, 
to do something about, and at what level”. 


Changes Implemented 

Participants were asked to identify up to five changes that were made to improve student experiences, learning, 
retention, and/or completion because of the program review process. Many changes were mentioned, though 
most fit into the categories of staff positions, student support, utilizing data, curriculum review, and assessment. 
A full list of changes implemented can be found in Appendix G. 


Staff Positions 

Several participants mentioned new staff positions that were created to assist with recruitment and supporting 
students. One institution discussed the addition of a CTE retention specialist to address the Perkins completion 
metric. The same institution identified the need for intrusive outreach for their students who were enrolled in 
technical programs and hired a staff member to assist students with navigational items such as technology, 
location of classes, registration, and scheduling. Other new staff hires included a part-time accommodation 
coordinator, additional instructors, and an online retention specialist. 


Student Supports 

Participants also shared a wide range of student-support programming that was implemented. One institution 
shared that it identified a need for mentoring within CTE programs, specifically for men and women of color. A 
participant from the institution said, “We not only established the men and women of color initiatives but also 
expanded in-class peer mentoring into CTE to address the academic need and the non-cognitive and soft-skill 
need that we saw as deficient in our students based on data”. 


Another participant discussed how program review helped EMT faculty determine the students who were 
having a hard time progressing through the program. Data indicated that those who performed well were 
students who were legacy students (e.g., parents were in fire service). As a result, the program implemented 
student supports that taught the non-legacy students things they would not have normally learned in their 
instructional programs. 


Another participant shared that their institution moved tutoring services from student services to academic 
services, later breaking down tutoring services into designated centers for specific disciplines: “Now we have 
between 18 - 20% of students who go to academic supports...just by breaking that Walmart model down into 
boutiques. That has been an amazing result.” 


Data 

Participants’ discussions also highlighted the use of data to make more data-based decisions. A participant 
shared their efforts in trying to make decisions based on data: “With placement testing, looking at the data. Do 
we have anything that predicts success? Do we have anything that is arbitrary that is keeping students out based 
on nothing, based on something that is not predictable?” 


Another participant expressed a concern about special populations, outcomes, and equity issues: 
One of the things I’m thinking about coming out of program review this year is where do our students 


come from, who ’s attending, what schools, and what programs? Is it equitable? Are they getting to the 
programs they want? 


Curriculum Review and Development 

Several participants discussed reviewing their curriculum offerings. One person reflected on reviewing 
curriculum for program review: “Do all these courses fit? Do your electives need to be recouped or required? 
And what programs are now doing AAS programs? Are we saying we want them to take political science and 
not just any social or behavioral course?” 


Another participant shared that their institution has begun to look at how often courses were offered and 
discovered “‘orphan” courses, which prompted a review of classes to avoid addressing courses in program 
review that were obsolete. 


Reflecting specifically on curriculum development, one participant shared how an assessment on humanities 
education identified that students were not fully aware of communities outside of their own, leading to the 
development of a service and learning experiential program. Because of student participation, there was a 
noted “increased level of course success and retention based on the students being able to be immersed and 
going to attend a performance or visit the Art Institute of Chicago or something that related to service in their 
community.” 


Assessment 

In reviewing its assessment process, one institution shared how producing enhancements led to faculty buy- 
in and made the process more valuable. “The forms that faculty use and the timeliness or when reporting 
requirements are there,” said one participant. Elaborating, this person added: 


They have developed a more systemic approach to assessing not just the programmatic goals but the tenure 

cycle of how the core competencies are going to be done, and whether those core competencies are going to 
be done through an institutional-wide process in one year, and then in the next year the expectation is that it 
is done at the programmatic level that feeds into the institutional data. 


Disaggregated Data 

Part of the session on equity focused on data that are collected as part of the program review process. 
Specifically, participants were asked to share the types of data reviewed to include subgroups of students 
explored. Responses indicated that institutions reviewed a range of data. One particular set of data explored the 
enrollees in specific courses, reviewing the seat count of those enrolled in the courses, as well as completers 
and non-completers. Enrollment, completion, and course-outcome data were also offered and disaggregated 

by those who withdrew and those who were successful. The data also revealed the time to completion, gender, 
race/ethnicity, age, first-generation status, enrollment status (e.g., part-time vs full-time), parents’ highest level 
of education, developmental education placement, and course- related items (e.g., section mode and location of 
course). 


Another participant added that their institution disaggregated enrollment, completion, and course success based 
on Al, A2, and AC data. Other ways in which data were disaggregated or reviewed included cohort completion, 
costs and revenue, the pass and fail rates of certificate exams, course modality, labor-market data, and denial 
rates for those who applied for graduation but were denied. 


Lessons Learned 

Participants were also asked to share what they learned from the data. Several institutions agreed that their 
communication and recordkeeping were poor. Other institutions discovered the need to terminate programs that 
were not being utilized, as well as the need to improve data collection. 


A few participants identified that differences in definitions of success has caused issues in the program review 
process. For example, one institution shared its trouble with the definition provided by ICCB: “Without having a 
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definite definition, we’ve inflated success. Our faculty are looking at our standard way of defining success when 
describing results and transfer rates based on that instead of what’s in the workbook,” said a member of the 
institution. Another participant stated, ““We’ve had to recalculate success.” 


Several participants spoke about the need to work on CIP codes. Specifically, the discussion focused on the 
relationship between CIP codes, stackable degrees, and certificates. One participant focused on how students 
identify: 


When we do a program review, we organize program review by the CIP codes. So then, there is some 
duplication in the numbers, because if these courses are required for this certification are also required 
for this AAS. They are getting the student counted in both places. 


Discussing Inequities 

Lastly, we asked participants if they discussed the discovered inequities within the programs of their respective 
institutions. Responses to this prompt were limited to only one focus group session, as all the facilitators 

were not able to progress to this question. One response indicated that people did not understand it. Another 
participant shared that faculty seemed to take discussions of inequity personally, “tying their value and their 
worth to the results.” One participant provided an example of discussing the gaps between male and female 
students and faculty saying, “I treat everybody the same” or “I’m not talking about that.” 


What is clear from the participants’ responses is that more work needs to be done to engage staff and faculty in 
having conversations around inequities and ways to tackle them without internalizing the issue. One participant 
stated, “That is a big question mark for me as I work with faculty to determine what the strategies are, the action 
steps we will take. They are struggling with separating their value from the data.” 


Infrastructure and Information Technologies 


Having the proper infrastructure and technology in place is vital to the success of program review. In looking at 
infrastructure and information technology, this section focuses on the systems that are available to support the 
collection, storing, access, analyzation, and dissemination of data throughout the institution. The discussion on 
infrastructure and information technology was guided by two questions: a) what role do IT resources have in a 
college’s ability to successfully conduct program review; and b) in what ways do the IT resources at a college 
support or not support efforts to effect evidence-based change? 


Role of IT 

When asked about the role of IT in colleges’ ability to successfully conduct program review, participants 
focused primarily on functionality. Examples provided include data integrity, storage, accessing data, 
practicality, and feasibility. Participants were asked to further reflect on how data is accessed. One person 
discussed the importance of having definitions and said “you need to know what you’re asking and the 
implications of what you’re asking, or talk to somebody who does.” 


Participants were asked to discuss specific technologies and the role they play in supporting the institution 
through program review. Their remarks identified several technologies that included, but were not limited to, 
the use of collaborative tools, job forms, and third-party services. While OCCRL does not endorse specific 
technologies, software, or third-party service, a few of those mentioned by participants are highlighted below 
to show the variation in those used. Some of the key aspects about these technologies are expressed by the 
participants of the studies. 


Access as a Collaborative Tool. One participant discussed the use of an Access online tool, which was used to 
encourage collaboration in the program review process. There were several tabs for various assessments within 


11 


the program, and the participant shared that the use of the tool was beneficial as assessment, was a shared 
responsibility, and there was the ability to see what other departments were doing via their assessments. The 
participant shared: 


They could go back at any time and revisit the action plan annually with the dean. So that made it easy 
to work with and also accessible to anybody. The department chairs had edit rights and everyone else 
had view rights. So if you are working on your department review and you're wondering what biology is 
doing, you can go and see those things. 


Another participant, from the same institution, added: 


There is a philosophical thing about making it accessible because it is not just the department chair. 
There may be one or two other people who have access ... but the idea is that the assessment is a shared 
responsibility ... we are putting it on display for the entire college. 


Job Forms as an Online Format that Draws From Multiple Datasets 

Another participant shared the use of job forms and their ability to serve as an online format for the program 
review template. Utilizing this technology, the participant’s institution was able to have one place to find 
everything that was needed for program review. According to the participant, job forms “link out to an action 
plan and assessment plan. It links to Dropbox for the datasets and links to our data warehouse.” 


TK20 (Watermark) to House Multiple Different Nested Assessment Tools 

A participant shared their use of the newly implemented TK20 platform, which at the time of the focus group 
had not yet been updated to migrate to the ICCB program review template. However, the participant said “our 
entire programming, our strategic planning, academic assessment, and organizational outcomes assessment, 
are all now on TK20.” The participant shared several benefits of utilizing the platform to include having a 
dedicated space where faculty members could collaborate, look at other assessments, and work together across 
departments and positions. 


EMSI (Economic Modeling Systems International) to Integrate Labor Market Data into Program Review 
Referring to economic data, one participant discussed the use of the third-party software Economic Modeling 
Systems International (EMSI). The participant said the system “provides economic data to us at the county, 
as well as at the regional, state, and national levels. It also provides program data comparisons from other 
schools.” Additional benefits mentioned include the ability to “produce forms that crosswalk between CIP 
codes and SOC codes so that you are able to get that analysis very quickly and up front.” Additional software 
mentioned was SPSS and Qualtrics. 


Business Objects to Customize Data Queries 

Business Objects was mentioned as another method of software to make the data accessible. It also allows 
the user to customize data queries and explore multiple data sets. However, the software is only accessible to 
those at the institution with permission and a license. Another limitation, as shared by a study participant, is 
the learning curve, given that “the typical local user across the campus needs to either learn and use Business 
Objects or put in a request to IT to pull a report.” 


IT Supports 

Discussion on IT support focused heavily on the challenges experienced in the program review process. A 
recurring theme focused on the issue of getting data out. One participant said, “That’s our biggest challenge. 
It’s helping people get the data out. There’s lots of data in the system, but people are reluctant to figure out how 
to get it out.” Another participant shared that data “is not available as quick as somebody who wants to make a 
decision.” 
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Another participant focused on the organization of the infrastructure: 


And there are people who have that skillset to do that slicing and dicing of data, and they are the ones 
that have the technical tools to do that. And then there are others that are consumers of the data. They 
have the need and questions, but they dont have the data skills to do the slicing and dicing, so they have 
to rely on the hierarchy. 


Similarly, another participant discussed the implications presented when staff members are not trained on data 
systems and have to ask for data: 


Because then you take four reports, run them all, merge them, hope you don t make a mistake in 
maneuvering data in Excel just to get what you're asking for. When a programmer could have gone in 
there, ran a report for what you’re asking for, and dropped it down. 


One institution shared an effective practice it implemented in bringing the data warehouse into the institutional 
research and planning office: 


We have the Banner module coordinators who collaborate with our data warehouse administrator. So 
we re getting to data much faster. We're slicing and dicing it much faster. In fact, with the volume of 
questions coming in, we can turn it around much faster. 


Leadership 


Formal and informal leadership play significant roles in the overall functioning of any organization. Formal 
leadership is often signified by codified power and authority, while informal leadership tends to be more fluid 
but includes facilitation efforts that allow for the completion of complex tasks involving diverse stakeholders 
within an organization. Because of the organic nature of how informal leaders arise within organizations, focus 
groups found it challenging to differentiate between campus “leaders” and “facilitators,” revealing that these 
identifiers were not mutually exclusive. One focus group participant commented on the role that seniority plays 
in informal leadership, saying, “Even if you go into a different role, you’re still viewed as the person with the 
most institutional memory, so the role follows you.” 


Example facilitation tasks that were discussed included creating data repositories, drafting reporting templates, 
organizing program review committees and panel discussions, and assisting faculty and staff in accurately 
reporting data (1.e., mentoring). Tasks and responsibilities associated with formal leadership centered on 
delegating duties, editing, and submitting finalized program reviews. 


The apparent interdependence between formal and informal leadership prompted one participant to state, 
“Leadership is everyone’s job.” This sentiment aligns with what Harris and Spillane (2008) refer to as 
distributed leadership, which is a perspective that “acknowledges the work of all individuals who contribute to 
leadership practice, whether or not they are formally designated or defined as leaders” (p. 31). 


Similarly, other scholars address how organizations can develop a culture of inquiry, whereby professionals 
seek to address pressing issues by considering available data. Leaders play a crucial role in developing an 
organizational culture centered on inquiry. Copland (2009) discusses how leaders can facilitate an inquiry- 
based environment by first understanding that they, too, are learners who do not know all of the answers. By 
distributing leadership roles, expectations, and responsibilities, organizations can foster a sense of co-ownership 
in which “the participants often become co-leaders and over time they develop shared norms and expertise in 
date-informed problem solving” (Copland et al, p. 159). 
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When prompted to consider traits of effective leaders, focus-group members articulated numerous traits that 
included effective communication, accountability, the setting of clear expectations, having a vision, and 
approaching leadership as a service to students, campus, and the community. 


Relatedly, one participant stressed the interpersonal qualities that are necessary for effective leadership, stating, 
“Leadership is appreciation. If you do not appreciate the people you’re working with, then you’re not going to 
have that teamwork, that vision. You appreciate people where they are and help them get to where they need to 
be, but you have to appreciate them.” 


Effective leadership traits that surfaced during focus-group discussions align well with “authentic leading” 
tactics identified by Pielstick (2000), which included guidance, relationships, character, shared vision, 
community, and communication (p. 104). Additionally, participants identified ways that institutional structures 
influenced not only formal position titles and related responsibilities, but also how tasks associated with 
program review were delegated across internal stakeholders. One participant said, “So, what we are seeing is 
that our organizational structure has a big impact.” 


Another participant echoed these sentiments, stating, “And that is what we found over here earlier: that we are 
all structured differently.” 


Furthermore, one participant provided an example of how external macro-level factors can adversely affect the 
program review process: “Many of my departments, particularly after the two years of budget crises, are down 
to one person, and they may have five or six different reports to write. That is tough; that is very tough.” 


Statewide Guidance and Governance 


One area we believed to be important to the program review process is the role of statewide guidance and 
governance. Such guidance and governance is defined as the state providing clear supportive standards and 
policies that support both accountability and data-driven change. Participants were asked to discuss their 
experiences during the program review process. Specifically, the discussions of participant experiences centered 
on the program review manual, ICCB support for the program review process, and direct support from ICCB. 


ICCB Program Review Manual 

Most participants indicated that they found the program review manual to be easy to use and relevant to each 
aspect of the program review process (Table 3). When asked to discuss their use of the manual, one participant 
said, “I think that provides a very nice framework. You can always improve on the template, but it’s the 


minimum expectation of standards.” There was a consensus among participants that the use of the template was 
helpful. 
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Table 3 
ICCB Program Review Manual 


Strongly j Strongly 

Merde Agree Disagree Disaeres 
Manual was easy to use 37 5% 78% 16% 0% 
Aspects of manual were unclear 38 69% 50% 24% 0% 
Manual was relevant to each aspect of process 36 11% 64% 22% 30 
Templates provided were helpful 38 11% 61% 24% 5% 
Questions posed in manual were relevant 36 80/ 580% 31% 30% 


Regarding the clarity of the program review manual, 76% of participants found aspects of it to be unclear, 
specifically in terms of the questions asked and the use of jargon. For example, one participant stated, “There 
are some questions I don’t get how I am supposed to answer. This question doesn’t make sense.” Another 
participant stated, “Is the data disaggregated by modality or something like that? Yes, it is. What really is the 
question?” 


Another participant focused on the use of jargon: “I was able to maneuver through the manual because 

I understood the jargon. I cannot imagine someone who has never touched a program review manual 
understanding what is really being asked for.” Several participants agreed, citing the need for a glossary of 
terms, explicit definitions, and samples of responses to questions. 


Participants also concurred that the questions posed in the manual were relevant (64%). However, they 
discussed the presence of inconsistencies in what ICCB requests for program review in contrast to what is 
accessible to the institutions. Specifically, participants discussed the request for labor-market data, but that there 
was no access to this information because institutions were no longer collecting it since ICCB was not requiring 
the graduate follow-up survey. 


While a majority of participants found the provided templates to be helpful, they identified issues with 
standardizing the process since it lacked the flexibility that institutions believe is needed to be more reflective 
of how data is collected and structured. Participants also identified the need to be able to adjust the template to 
include questions of interest to their respective institutions. 


ICCB Support 

When asked about what ICCB support participants found to be most beneficial, responses were mixed. A 
majority (38%) disagreed that there was adequate information regarding what supports were available (Table 4). 
Further, 34% of participants indicated that options of support were satisfactory. What is even more striking is 
the percentage of participants who responded to survey questions by choosing the Not Applicable option (Table 
4). One participant stated, “I didn’t have any support. I don’t know if I am on the right list.” 
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Table 4 
ICCB Support for the Program Review Process 


Strongly : Strongly Not 
Agree meres,” DISaEtee Disagree Applicable 
Adequate information on supports 37 0% 32% 380% 50/ 24% 
available 
Supports provided were helpful ay 0% 43% 149% 5% 38% 
0 
Options of support were adequate 35 0% 34% 40% 0% 26% 


Participants discussed their limited access to support as administrators served as gatekeepers of information. “In 
the past the associate dean and assistant vice president of student affairs is the person who interacts with ICCB 
and puts the report together,” said one individual. 


Another participant said, “We never looked at any of this stuff except for the template that got sent out.” The 
individual continued: “It’s not an open door. It is one person who has a key to the resources, and we go to that 
person and they facilitate all that.” 


In addition to limited access, participants shared that there was limited knowledge of what supports are 
available. It was suggested that if there are supports that this information may not be be shared beyond the 
primary contacts with ICCB. One participant stated, “I don’t remember any support until the webinar this year.” 


Table 5 
Direct Supports from ICCB Staff for Colleges’ Program Review Process 
Strongly : Strongly Not 
Agree eERee  Basvere Disagree Applicable 

ICCB staff were easily 37 14% 350 50/ 0% 46% 
accessible for support 
ese emecineee ie 14% 38% 3% 0% 46% 
responding to inquiries ail 
Information received from 2 é P ‘ P 
ICCB staff was helpful a ths eeu a ue Bt 
ICCB staff were 
knowledgeable of program 37 11% 49% 50/ 0% 350/ 


review process 


Participants were also surveyed on their experiences with direct support from ICCB staff during the program 
review process (Table 5). A majority of them agreed that ICCB staff were easily accessible for support (49%), 
and that the information received was helpful (49%). One participant stated, “The best resource from my 
experience has been actually calling ICCB and getting answers from them. That is the only resource other than 
the manual.” 


Similarly, participants responded positively to questions regarding the timeliness of responses to email and 
ICCB staff knowledge of the program review process (Table 5). However, there was also a large population 
of participants who indicated the “Not Applicable” choice to questions that pertained to direct support. This 
corresponds with participants’ experiences of support not being filtered down due to institutional gatekeepers. 
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Challenges 

In discussing the lack of support available, participants identified several areas in which guidance would have 
been helpful. One area of need is in the realm of disaggregated data. One participant stated, “I struggled on the 
disaggregate data. There just wasn’t any information there to help me define that.” The need for examples of 
questions that would lead to disaggregated data was also discussed. A participant shared: 


So when we talk about disaggregating data, we should let the data lead us to our questions. But I think 
it would be really helpful for faculty to have some examples. What types of questions should this be 
leading us to? Marketing? Gender? What? I think it would be very helpful for them to get their heads 
around what we mean by disaggregating data. 


Another challenge identified was the definition of a program. Specifically, participants discussed variances in 
what they considered to be a program and the need for clarification on what ICCB is looking for. In particular, 


participants related the terminology of major versus program, saying, “Some students have a major that requires 


them to get an associate degree and then transfer, and some don’t.” 


Participants also inquired as to whether stackable degrees are counted as individual programs, and if so, what 
the purpose is of counting them that way. “How does it help the institution to do separate program reviews on 
the program internally?” one participant asked. “What is the value of the report on the stackable?” questioned 
another. 


Five Design Challenges 


Using the focus-group data, the research team identified five challenges that influence the design of program 
review. The five design challenges identified were: 


Identifying who is enrolled in each CTE program 

Assessing stackable credentials and addressing concurrent enrollment 

Balancing workload and workflow due to the program review cycle and groupings 
Dealing with insufficient ICCB Support and Feedback 

Using disaggregated data to identify equity gaps 


ine Foo 


These issues were presented to participants as design challenges, and the results are summarized below. 


CTE Enrollment 


Accurately identifying who is enrolled in every CTE program is a critical challenge for colleges, and that was 
highlighted throughout the focus groups. This challenge creates a major barrier to the ability of institutions to 


use program-level data in their decision-making processes. If faculty and administration are not confident in the 


quality of the enrollment data, then any analysis based on this data is not reliable or trusted. 


As a result, the quality of this data is both functionally essential for the viability and reliability of any findings 


based on the information, but the trustworthiness of the data is critical for buy-in and use in the decision-making 


processes. There are many reasons that capturing enrollment information is a challenge for nonselective CTE 
programs. These barriers include, but are not limited to, the following issues: 


a) Assigning students to a program of study assumes that students in all fields intend to complete a program 


of study (e.g., not take specific courses for reskilling or are undecided). 
b) There are challenges in identifying and accurately reflecting students’ current and intended program or 
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programs of study, especially when their intended program changes during the course of their studies. 

c) There are unclear rules of exiting students from programs of study, and for distinguishing between those 
who have completed a credential (e.g., a certificate), changed programs, or left college. 

d) Students may be enrolled concurrently in more than one program of study in the cases of stackable 
credentials and dual majors. 

e) Students can concurrently enroll in more than one institution and either have the intent of earning a 
credential from just one of these institutions or of earning different credential from multiple institutions. 

f) Uncertainly whether to include or review programs that are designed specifically as pathways into 
CTE programs (e.g., pre-nursing, pre-radiology, etc.), as CTE, and challenges with determining both 
enrollment and completion measures for these programs. 

g) Due to developmental coursework, changes in majors, and coursework from other institutions 
(concurrent enrollment and transfer credits), it can be challenging to identify when students enter a 
program of study; and when using this data, students cohort. 


Responding to the Challenge of Capturing Accurate CTE Enrollment Data 

The response to this challenge varies based on the size of the institution, the types of programs, and to what 
extent the different challenges are impacting particular programs. Institutions recognized that because of the 
numbers of variables involved, the goal is to improve the overall quality of the data and, where possible, to 
operationalize it. 


Participants offered several strategies for improving program enrollment data based on practices at their 
colleges. For example, in terms of operatizing the data, it was suggested that the state consider aligning 
program review data standards with those utilized for other statewide data collection, specifically those captured 
through the state’s Annual Enrollment (A1) Records. Ensuring that the analysis done based on enrollment and 
completion is accurate and comparable, even at the institutional level, means being clear about what the specific 
timeframe is, who is included, and who is excluded (e.g., exited). 


Utilizing CTE-specific advisers was another strategy, in an effort to have someone who is familiar with the 
programs and required coursework, and who can work closely with faculty and staff to monitor the alignment 
between students’ declared majors and their course enrollments. 


Similarly, participants reported that they were able to improve their data by increasing the regularity of meetings 
between advisers and students, either by requiring students to meet with advisers at set points in their program 
or through a regular schedule during the semester. One participant shared that at their college: 


Students see an adviser at the 19th and 41st credit completed. The system automatically puts a hold on 
their records if they dont get in and see an adviser. So that’s something systematically that could be 
done to make sure students are on track. 


Further, these advisers can identify students whose major does not align with their course- taking patterns by 
using degree-auditing systems for the purposes of establishing eligibility for financial-aid supports. To that end, 
one participant said: 


There ’s a financial-aid model that all financial-aid departments have to look at. They’re only going to 
pay for courses that are required by the program, so we had a recommendation that we should expand 
that model across everyone, and basically doing degree audits regularly so we can pick up when 

a student is either taking classes in a different major or they’re still on track, and make sure we’re 
following them. 
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Additionally, participants suggested it is helpful to associate majors with student names on course rosters for 
faculty. Doing so provides valuable information to faculty about the composition and intent of the students in 
their class, and allows the educators to help flag cases in which there appears to be a misalignment between 
students’ coursework and their intended major. 


Participants suggested reviewing and streamlining the processes and procedures used for updating students’ 
major codes in the colleges’ systems. It is important to ensure that the process protects students who may be 
unaware of how changing a major may affect them, especially in terms of financial aid. One participant said that 
at their college: 


We synchronize that by term, so after the census date, you cannot change your program. You can make 
the request. It will change at the next term date. So the aid is awarded based on that, and that aid stays 
stabilized based on your program for that term. And then we run that course-applicability process. 
Every day we run it, and synchronize it with financial aid. So we won t award aid on courses that don t 
apply to the program of study. 


Finally, participants felt it might be beneficial for ICCB to assist in developing alternative pathways for 
pre-CTE programs that allow them to be assessed separately from the transfer students also enrolled under 
Associate of General Studies programs. A participant shared: 


We put our pre-rad and pre-nursing students in Associate of General Studies (AGS). Which when 
reported to ICCB is considered a transfer, but these students’ intent is a CTE degree. So, we thought it'd 
be a good idea if ICCB could create a preprogram that we could use for reporting that as a pre-CTE, 
because if those pre-students are unsuccessful and then dont end up going into that CTE program, thats 
a failure at CTE and not transfer, and they’re reported as transfer with the AGS. 


Addressing the challenge of accurately capturing CTE enrollment will require changes and efforts by both 
colleges and the state. However, the responses provided here provide a number of realistic strategies that can 
greatly improve the overall quality of the data, and improve the ability of colleges to produce reliable and viable 
program review assessment and evaluation data. 


Stackable Credentials 


Conducting program review assessments on stackable credentials and other closely related programs poses 
unique challenges for colleges. Specifically, it can identify students as enrolled in a particular program of study 
when they are concurrently taking courses in more than one program, especially when these programs are 
stackable. 


Secondly, many certificates in stackable credential sets are embedded in the associate program. This makes 
evaluating them separately from the associate program challenging and results in colleges reporting the 
same assessment findings across multiple programs. This practice in the realm of stackable credentials is not 
productive for assessing the value of the individual programs. 


The participants discussed the assessment challenges created when students are concurrently enrolled in 
multiple programs of study and those unique to assessing stackable credentials. The dialogue included exploring 
the impact of grouping credentials together for program review and what programs could successfully be 
assessed together. How can the program review process work more efficiently and effectively for stackable 
credentials? What information is important to know about stackable credentials outside of the individual 
program assessment process? How could we accurately capture and reflect students who are concurrently 
enrolled in multiple programs? What would the value of this change be? 
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Responding to the Challenge of Stackable Credentials 

Participants expressed that it is increasingly important that we work toward having systems that reflect multiple 
concurrent program enrollments for students. To support this for program review, participants suggested that the 
state should adjust its existing data collection to allow for multiple enrollments (e.g., annual enrollment records) 
and draw on this information for program review. One participant said: 


And the biggest limitation that we have, that I see with the A1, is that you can only report one program. 
So if you’ve got somebody who is maybe saying they’re in AA or something rather generic, because 

they re after aid, but they really want some other program. Or they’re in multiples. You can t really 
discern that. And I think it’s odd, personally, now that we have the A2 for the completion side, where you 
can report multiple completions. Obviously that means people are enrolled in multiples at one time. It 
seems like there should be a hybrid of the Al, A2 that allows for the multiple programs. 


Participants recommended that stackable credentials be assessed as a group, which could work in one of two 
ways. The first would be that certificates stacked under an associate degree would be assessed concurrently with 
the associate degree. The second is that certificates of different lengths would be stacked together (e.g., a short- 
term certificate that stacks with a long-term certificate) but are not linked to an associate degree that is bundled 
and assessed together. Participants clarified that even if multiple associate degrees share substantial amounts of 
coursework, they should not been seen as stackable and always be assessed separately. 


Participants emphasized it is essential that when stackable credentials are assessed as a group, questions that 
access the quality, cost, and need for each credential in the stack be included. This could be done by adding a 
separate template for stackable credentials or integrating a set of questions to the existing template that identify 
any stacking credentials where applicable, and then including a series of questions that are only answered for 
stacking credentials. Specifically, the participants suggested that, at a minimum, the questions should cover the 
following: 


a) What is the rationale for any certificate in the pathway? 
b) Does each credential in the pathway hold industry value (e.g., improved employability)? 
c) Is each credential independently viable? 


Participants described a pattern of creating credentials that are recognized by industry and may hold limited 
value for students. One participant explained: 


There are two things for colleges to do; one of them is to clean up what we’re lovingly referring to as 
‘garbage certificates.’ You have these certificates that do not necessarily have market value. They may or 
may not articulate to anything.. 


Program Review Cycle 


The five-year cycle of program review presents challenges for institutions because there are five years between 
reviews, and the grouping of the programs are reviewed in each of the five cycle years. Yet while most 
institutions were clear that a five-year cycle is too long to be effective, they found that other challenges related 
to the cycle and grouping of the programs vary greatly based on role and institution. For example, while some 
colleges expressed that the grouping of programs sometimes results in a small number of staff having a large 
burden of programs to review simultaneously, this same phenomenon was viewed by other colleges as a barrier 
and by still others as an opportunity for cross-disciplinary collaboration and setting shared strategies across 
programs. 
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The participants discussed strategies for optimizing the schedule of program review. Specifically, they grappled 
with how to capitalize on the desire of colleges for more frequent reviews, without making the process 
unfeasible and overwhelming, especially for those with limited staffing and resources. They also contemplated 
alternative grouping of programs within the existing five-year cycle and what ICCB could do to help colleges 
overcome some of the issues created by scheduling. 


Responding to the Challenge of the Program Review Cycle 
Overall, the participants recommended maintaining the five-year cycle while creating the support structures 
necessary to encourage institutions to review the process annually. The respondents said as a group: 


One of the things we ’ve determined is that the five-year works with the multiyear analysis steps built 
in. Some sort of an annual review framework and some guidelines for that would be helpful, but it was 
determined that we still need the five-year cycle. Likewise, the annual goal review should be built in so 
that we can address the intended action items and review them and determine the use of data to report 
out regarding the program review. 


We think that ICCB should recommend an annual process but not require an internal process. ... Just 
being able to say ICCB recommends this, it can help them in doing that, but doesn t dictate what that 
looks like and doesn t require a formal report to the state. 


The participants expressed that by promoting an annual process, institutions could spread some of the work 
associated with program review over the five-year review period. This in turn could reduce the impact of having 
a program review cycle in which an administrator has a large number of programs under review due to the way 
the programs are grouped. 


The participants did not propose specific changes in how the programs are grouped for program review, except 
in that programs that are stackable should be reviewed together. This could be accomplished either through a 
review of the applicable CIP codes or through a waiver approved by ICCB. 


ICCB Support and Feedback 


The need for professional development and other supports was a common theme throughout the focus groups. 
Suggestions on the types of professional development needed, the topics, and the intended audiences were 
diverse among the participants. Likewise, colleges expressed a desire to receive feedback from ICCB on the 
completed program reviews and to receive from ICCB information about promising practices and benchmarks 
based on what is submitted by colleges statewide. 


Participants discussed the different forms of needed professional development, potential audiences, and the 
best formats to deliver these activities. They also explored potential forms of support that ICCB could provide 
colleges to foster effective program review processes as part of a culture of evidence. Finally, participants 
explored what should be included in the feedback from ICCB and what form this feedback should take. 


Responding to the Challenge of the ICCB Support and Feedback 

Participants indicated professional development is needed that is accessible to and directed toward the different 
stakeholder groups who are engaged in program review. They said priority should be given to professional 
development that supports program coordinators, faculty, and anyone new to the program review process. 


Further, it was suggested that modularized training, recorded videos, and other resources that can be accessed 
and used as needed would be preferable over time depending on professional development options (e.g., 
workshops). Participants also felt that job aids such as rubrics could be used to helps support internal feedback 
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and professional development, as well as serve as the structure for feedback from ICCB. However, it was 
emphasized that any such rubric should focus not on the content but on the quality of the response itself (e.g., 
how well did it address the question?). A participant summarized: 


That was basically for the use with the ICCB feedback, perhaps they could establish a rubric that would 
be shared with all the colleges, so that you would know some items and what the various criteria would 
be for “emerging, meeting, and exceeding” on that rubric. 


As for other feedback from ICCB, colleges expressed a desire for program review to support college-to-college 
networking and sharing of promising practices. It was emphasized a few times that based on how the programs 
are grouped, the sharing of practices could be classified by discipline area. A participant suggested: 


Giving the constructed feedback would hopefully include perhaps extracting, for example, “what 
innovation does your CTE program offer,” so they [ICCB] could extract that...and do some summary 
Statements. 


Another participant suggested: 


In addition to getting feedback to the individual college about their report, and how well they did if there 
could be, and I think they did this a few years ago, that there would be a statewide report. Or at least 

if they asked about innovations and, just copy and paste everybody's answer to that so you could read 
here's what all the automotive programs innovations... That we are all giving ICCB this information so 
what would be a way for ICCB to make it available for everybody. 


Additionally, participants suggested that ICCB could help build connections between programs to foster 
mentorships and networks as needed, particularly for the purposes of connecting new hires or linking struggling 
programs to mentors from strong programs. Lastly, the participants shared that they look to ICCB to help them 
stay abreast of important policies, practices, and research from outside of the state. 


Disaggregated Data and Equity Gaps 


The new program review manual includes the analysis of disaggregated data and identification of equity 

gaps for CTE programs. Further, colleges are asked to explain what they are doing to address gaps, and if the 
students in the program are representative of the college, population, and the college’s district population. 
Several colleges shared that they were unprepared to be asked these questions and found this portion of the 
program review process challenging. Participants explored how the program review process could help colleges 
to identify and address educational inequities in their programs. Finally, participants discussed how ICCB could 
support colleges’ efforts to identify and address inequities through the program review process. 


Responding to the Challenge of the Disaggregated Data and Equity Gaps 
Context plays a critical role in the ability of programs to identify and address inequities. Participants felt that 
equity needs to be addressed strategically at the institutional level. One participants explained: 


We started with, maybe it was from a strategic plan, but then we said we can t just assume that all 
strategic plans have those bullet points within it. So at least the institution can contextualize that, and 
then have the programs address it. 


Identified inequities at the discipline or institutional level are often more actionable then at the program 
level, especially when dealing with programs or disciplines with relatively small enrollments over time. One 
participant said: 
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If the institution can identify, is asked to identify what are the educational inequities within the 
institution, that puts it in the right framework for each of the programs to respond to because we’ve done 
something institutionally with it. And that pushes it, with that kind of a question, which is outside of each 
of the individual templates, that pushes the institution to do that and have that discussion. 


The current process does not allow institutions to address equity within these or the other relevant contextual 
factors (local labor markets, secondary systems, for example). Program-level inequities can be better understood 
and acted upon when seen within the context of their relevance to the context in which they are situated. 


Additionally, participants highlighted that relying solely on quantitative data, especially in small programs, as 
indicators of equity and inequities is limiting, and that a more balanced approach may reveal added nuanced 
inequities and more readily lead to effective resolutions. 


Participants indicated that it is critical that the results of this section are actionable. They suggested that either 
there should be a clearer delineation of this section of the program review process as focused on equity (versus 
data), or that the questions be embedded into the relevant sections on quality and need. Relabeling the section 
as equity would be more transparent and allow for qualitative questions that would let programs explore their 
context, relate it to the discipline and college level, and identify necessary action steps. 


Participants indicated this work is outside of the knowledge base of many of the stakeholders participating in 
the program review process. As such, they suggested that clear definitions of equity and inequity are needed, as 
well as professional development that covers both identifying inequities and addressing them. 


Finally, participants considered the role of ICCB in the collection of this data, what ICCB intent is, and how 
ICCB could support its efforts to address inequities. One participant said: 


So when we started talking about the ICCB and what they could do ... One of the things we brainstormed 
about was whether or not there is ... If ICCB is looking at this from a system-wide perspective, are 

there system-wide issues or inequities they are identifying in the system? Could those be identified and 
institutions could respond to that? While they might not be experiencing the same inequity within the 
same institution, they could address that upfront, “and this is why, but we have all these other inequity 


issues.” So, just what is the system looking for? 
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CTE Program Review Template 


This section of the report focuses on specific areas of ICCB’s CTE program review template and offers 
suggestions for improvement. 


Indicator 1: Need 


Focus groups were asked to review how questions are arranged and worded for Indicator 1: Need. Questions 
1.1-1.3 require access to high-quality, regional labor market data. While the budget of some colleges allows 
for greater access to this data, other college budgets do not. Some participants felt that sections 1.4 and 1.5 
would benefit from being combined into one question that differentiates between institutional recruitment 
(e.g., marketing department) and program recruitment (e.g., faculty-led career fairs or demonstrations at high 
schools). One participant expanded on this recommendation: 


What we were saying is those three questions are so related... it might have been more helpful to put it 
in a paragraph format so that it gives it more context and they can just talk about occupational need 
based on each one of those questions, and then referring to it both with the actual data from either the 
Bureau of Labor Statistics, but also use their anecdotal data, which might come from Career Builder 
or might come from a number of calls that they’ve received from employers asking for students or 
graduate information. 


Furthermore, colleges were unsure of which institutional activities were outside of the immediate program 
(e.g., recruitment, marketing) that they should or should not include in these reviews, especially in the “Need” 
section. 


Lastly, participants requested access to a copy of the previous report in the cycle to compare occupational 
outlooks and trends. The archived review could serve as a reference for the current reporting cycle. 


Indicator 2: Cost Effectiveness 


Participants were asked to review the questions for Indicator 2: Cost Effectiveness. Specifically, they were 
asked to provide suggestions for improving upon the provided questions. The first suggestion focused on the 
term costs. Participants shared that programs struggle with how to define and calculate costs. Given there are 
multiple levels of costs (i.e., return on investment, department, institution, and student), clarification is needed 
regarding how ICCB determined cost effectiveness. One participant stated, ““That’s where we struggle. We 

need definitions. So what should go into that cost? Break it down, what should be part of that total for each 
program?” Another participant shared, “Because we talked a lot about unit costs, indirect costs, salary costs, and 
to the point that the gentleman over here made, not all of us have that, are privy to that information.” 


Relative to the issue of defining cost is how to assess costs without balancing revenue. This came up when a 
participant discussed the “invisibility” of how much revenue from the programs are due to general education 
courses, and the difficulty they pose to measure costs accurately. The participant shared their experience: 
“Because they’re in an automotive course or program or welding program, they still have to take math. Still 
have to do the humanities. All that revenue that they’re bringing in offsets the costs, but it’s really hard to 
measure.” 


The elimination or revision of several questions was also suggested. For example, one group said that the 
question asking how costs compare to other programs (2.2) should be eliminated, since costs are not comparable 
across programs, and the question was not of value to the institution. However, another participant advocated 
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for its inclusion, providing an example of how their institution made cost comparisons: 


When we get to the comparisons to other programs on campus, one of the things we talked about was 
looking at the manufacturing programs. I might compare welding to the industrial maintenance program 
because they’re both in manufacturing. But I wouldnt compare welding to business or welding to health 
professions because there's variances in the costs there. 


In terms of rewording questions, participants had several suggestions. Question 2.4 asks specifically about grant 
funding. One participant suggested changing the wording to remove “grant” and including “external,” saying 
the question could read, “So if external funding went away, would it [the program] still be sustainable?” 


Several participants discussed the wording of question 2.5, critiquing the question in its current form as 
soliciting a “‘yes” or “no” response that held no value to the institutions. Instead, participants recommended 
rewording the questions in more action-focused ways. These suggestions included: 


a) How do you plan to modify the program based on costs? 
b) What changes or modifications were [or will be] implemented as a result of the program review? 
c) What actions or steps did we [or will we] take as a result of the program review? 


Participants also made suggestions regarding questions that institutions added to their individual program 
review templates that were not initially included: 


a) Are you anticipating any additional or unusual costs in the coming years? 

b) Are there external factors that impact the cost of the program? Has the cost trended over time? How is 
that trend looked over that five years? Has it gone up, gone down, or stayed steady? 

c) What strategies do you have to mitigate costs? 


The use of program review in the budget process at institutions was also mentioned. For example, one 
participant reflected on future budgeting for her faculty needs. She suggested an additional question about 
anticipating budgetary needs. “What, if any, specific budgetary needs do you anticipate having over the next five 
years?” she asked, and then went on to say: 


We ask them, “What anticipated costs do you have?” And they might say, ‘We want to buy this machine 
but we need to plan for x, y, or z, so we get the things out there because we then tag this to our budgeting 
process.’ When the budget office says, “I need a hundred thousand dollars to buy this printer,” they will 
say well it was in their program review. 


Indicator 3: Quality 


In reviewing participants’ responses to Indicator 3: Quality, it was clear they did not feel the questions provided 
in the template spoke to the quality of the program. Instead, participants’ reflections of the questions indicated 
that they read more like a checklist of components. One participant stated, and many agreed, “They have 
nothing to do with quality and the impact.” 


Question 3.5 asks about innovations that were implemented. A participant shares their thoughts on the question: 
What does that have to do with it? If I don’t have an innovation, does that mean I am not a quality program 


and does the questions really share an effective practice used by the college to deliver instruction and support 
students? 
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Another example provided by a participant occurred with question 3.7, which asks about work-based learning 
opportunities. The participant offered an example of how the question could be rephrased to provide value: 


I shared an example of how we’re reframing that. We say: Describe how you provide students the opportunity to 
apply their skills in a workplace setting. Provide details about when those opportunities occur in the curriculum, 
and how that experience is designed and/or evaluated. That gets to the quality. Just the fact that you have them, 
what does that mean? Is it designed early on, midway through the curriculum, at the end? Is there a continuation 
of it? How do you evaluate it and drive improvement there? 


Other questions of consideration included 3.6, which reflected on the absence of dual-credit opportunities, 
which suggests a program is not of quality. A suggested rewording was offered: “How do you ensure the quality 
in the delivery of the courses in a dual-credit environment?” 

Another question, 3.17, asked, ““What assessment methods are used to ensure student success?” Similar to other 
questions, participants inquired as to what the question was getting at. 


Questions 3.8-3.13 also posed an issue for participants: 


We’re really trying to figure out what are we talking about, when we’re talking about industry accreditation, or 
recognized credentials in relationship to quality? Is it an apprenticeship? Again, it could be a descriptor, yes or 
no, or maybe the question really is about the delivery of the curriculum or the design of the curriculum. Is there 
an opportunity for it to be designed as an apprenticeship program? What value does the current articulation of 
cooperative agreements have or bring to the program or the student’s ability to gain employment? Just in that 
whole area there. That if it’s not an accredited program doesn’t mean that it’s not a quality program? 


Participants also discussed the need for operational definitions of key terms or components. Several of them 
agreed that the “faculty to student ratio” posed in question 3.14 was confusing and did not speak to quality. One 
participant shared: 


This is confusing. We all had different definitions. Some of us just wanted to eliminate the question. Some 
of us wanted to know how are we measuring or what’; the formula for the faculty to student ratio. Some 
argue using just an institutional student-faculty ratio. 


Another term unclear to participants was “innovations,” which they felt was vague. They also were not clear on 
ICCB’s intent in including that question in the template. 


Another area of frustration focused on student satisfaction. Participants indicated that, due to challenges in 
obtaining student response on the program level, many included institutional student responses, and that this 
limited the effectiveness of the institutions’ ability to respond to the prompt. 


Question 3.21 was also noted. It asked participants how often their program advisory committee meets. They 
again indicated that the question, as written, did not speak to quality. One participant shared how her institution 
rephrased the question: 


We ’ve rephrased it to describe not only how often the program advisory committee meets, because 
that’s what the question is, but also the recommendations that have positively affected the growth and 
improvement of the certificate and degree over the last five years, and describes the role of the advisory 
committee, and the review of this program. 
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There was a brief comment on how program review does not capture any changes or intended changes to the 
curriculum, the quality of that curriculum, or the associated learning outcomes. One participant shared how his 
institution addressed curriculum to the template used: 


We have a series of questions that ask the department to talk about curricular changes, specifically 
changes to their course reference forms over the past five years and why they did those. What was the 
rational, and then, for one that they did not do, why did they not do those and what their rational? 


The review showed that there are disconnects within the program review process, curriculum, and student 

learning. Further, data and data analysis was shown to focus on the disaggregation of outcomes of learning 
at the course and program level, not on disaggregation of learning outcomes. This poses the questions: Are 
students learning in these program? And who is learning and what are they learning? 


Data Analysis 


Participants felt that the quantitative questions in the Data Analysis section should be in the first section of the 
CTE Program Review and should include a wider variety of college-level data. Rearranging the sections would 
allow for referencing the data already presented when discussing program-specific objectives and outcomes. 


Moreover, some felt the questions should be embedded within other sections of the report. It was mentioned that 
the Data Analysis portion should more accurately labeled with the term “Equity” since each of the six questions 
address student representation and access. 


Participants described the question “What is the college doing to overcome any identifiable gaps?” as too 
narrow. Many felt this question could be improved by addressing initiatives enacted on the program level and 
showing how the program is supporting colleges’ efforts to address equity gaps. 


Participants also expressed concern that many faculty were not skilled at identifying or addressing equity gaps; 
therefore, questions should be worded using understandable terms. It was noted that while faculty members 
are experts in their respective fields, participants felt that more guidance was needed for faculty to thoroughly 
and adequately answer questions pertaining to issues of equity. One participant said: “Maybe some operational 
definitions of some of the terms might be useful to help faculty to better understand what they [ICCB] are 
looking for.” 


Format of the Template 


Focus-group participants shared many ideas that were aimed at improving the program review template. Some 
stated that the current static PDF format is challenging to work with. They requested that future templates 
offer more functionality to improve responsiveness and ease of use. Participants discussed the desire to work 
collaboratively while drafting the document, which would allow for multiple stakeholders to make “live” 
comments simultaneously. Collaboration would also allow for the sharing of best practices throughout the 
college as well as with other colleges. 


In addition to sharing best practices, a responsive template would permit users to openly discuss challenges and 
delete these comments prior to the final submission. A shared template would contain the functionality to allow 
levels of permissions for editing, sharing, and more. Participants requested that the proposed template accept 
links to artifacts that could then be shared with ICCB, with links to sources being embedded within the report. 
They recognized that changes to the program review template should be piloted before final implementation, a 
process that would allow institutions to preview changes in the field and quickly address any issues. 
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Advancing Program Review 


We asked all participants in this section to work in small teams and envision future changes in program review 
in 2022 and beyond. These teams of participants highlighted one aspect of program review that should change 
and the supports necessary to support that update, as well as one aspect of program review they felt should 
remain unchanged. Below are their summarizing thoughts. 


“Concise is Nice” 


During the focus-group portion of the study, participants brainstormed and reflected on ways to make the 
program review process more user-friendly for stakeholders across their respective campuses. Working 
groups were asked to consider what they would need to maximize the program review process. Following 
deliberations, groups reported the need for an easy-to-use program review toolkit. One group succinctly stated 
how it envisioned this toolkit: “Concise is Nice.” 


A concise toolkit would describe clearly what is expected of each program under review. Participants spoke of 
the creation of a data dictionary to assist colleges in operationalizing variables being measured. The frequently 
mentioned the lack of standardization within, and between, institutions, with one participant saying, ““We need a 
base to start with, so we can all have a shared understanding.” 


A shared understanding is more likely to be cultivated if all colleges have an agreed upon definition of what 
variables they are measuring and how they are measuring them. Likewise, participants requested a glossary 
of terms to supplement the data dictionary. And the data dictionary, meanwhile, could include descriptive and 
illustrative examples or rubrics to help foster clarity and consistency of reporting. 


Participants felt a thoughtfully constructed program review toolkit would help institutions build the data quality 
necessary to benchmark across institutions. In this way, Institution A could compare student-learning outcomes 
and enrollment data with Institutions B and C. Only until these institutions have “a base to start with” would 
their comparisons be reliable. 


Finally, participants discussed the importance of communication during program review, while focus-group 
members felt it was necessary that all communication be clear and happen early and often. 


“You Help US to Have the Important Conversations” 


To strengthen future program reviews, focus-group participants expressed the need for a cross-pollination of 
ideas among peer institutions, with the goal of highlighting the value of discussions such as those that took 
place during this research process and crediting ICCB for supporting these types of exchanges. One professional 
requested that ICCB “help us to have the important conversations.” 


It was noted that ICCB does not require institutions to share high-impact practices, nor are related practices 
recommended by ICCB. Participants recognized a communications gap that must be bridged to promote 
the sharing of ideas, with ICCB playing a facilitative role. Participants placed a high value on talking with 
representatives from other colleges during the three focus group meetings. These gatherings promoted 
discussions on program review templates, reporting, and ways in which others addressed reoccurring issues 
such as access to data and stakeholder buy-in. The focus groups also aided participants in developing social 
capital that could be tapped for mentoring and/or troubleshooting. 


“Leverage the Schedule and the Process” 


Regarding the program review cycle and process, participants wanted more time to prepare for the 
implementation of changes to the process. One stated, ““We need the time to have accurate planning. We need 
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the time so that it’s well thought out, it’s a plan, it’s vetted, and it’s not just rolled out in a hurry, and then we 
need to be more proactive as opposed to reactive.” 


Participants agreed that the program review process should be more continuous and integrated within 
institutions’ regular curriculum processes. An annual update for all programs was suggested in this realm 

as one enhancement. Several participants also discussed the five-year cycle, agreeing that maintaining that 
timeframe was important. In doing so, there could be a dissemination of best practices by academic discipline. 
A participant shared that the need for continuous improvement is important. Therefore, the five-year cycle is 
equally crucial, and having a standardized template that reinforces the idea of continuous improvement were 
things many would like to see happen. Another participant agreed, saying: 


An ongoing continuous model, not just every five years. In terms of one aspect we really liked. The 
schedule with the opportunity that we could, perhaps, leverage and benefit having all programs reviewed 
in the state at the same time. 


Aligning the program review process with HLC standards was also recommended. “It would be nice to have 
this process aligned with the same standards that we need to have met there,” one participant said. A few 
participants also brought up the timeliness of process in regard to the release of new materials. One person 
reflected on the abruptness of the most recent version and called for an earlier release of the next, saying, “They 
should have the next manual out in 2021 for the cycle that starts in 2022. It’s a whole year in advance, but we 
need that.” 


To summarize, participants recommended maintaining the five-year cycle, timeliness in the release of new 
manuals to allow for preparation and training, annual updates for programs, and aligning with HLC standards. 


“We Need Input from Our Peers” 


Participants acknowledged the depth of expertise that was present and felt that peer input was important to the 
program review process. One group shared: 


Basically, we all realized that we need input from our peers and that there is an untapped potential of 
resources in this room. We’ve all learned so much from each other, and it will help us all to get better 
every single year. 


Participants offered suggestions on how to incorporate peer input into the program review process. One group 
said a peer-review system would allow program review committees to receive feedback in a timelier manner. 


Similarly, another group said: 


We thought we should create a core group from this focus group to put together suggested changes, 
create the template, create the rubrics, and then have that core group pilot it, kind of get the kinks out. 
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Next Steps 


OCCRL intends to extend and expand, in collaboration with ICCB, on the information learned from the 
expertise shared in this initial study. This will include facilitating the development of a program review advisory 
committee and an online learning community. 


Program Review Advisory Committee 


OCCRL will collaborate with ICCB to establish a statewide advisory committee that meets quarterly to provide 
formative feedback to ICCB about the program review process, including feedback or suggestions on draft 
changes and the piloting of proposed changes. This advisory committee will be comprised of eight professionals 
with substantive experience with program review in Illinois, individuals who are currently employed at an 
Illinois community college and are willing to serve on the committee for at least two years. Three of the four 
quarterly meetings will be virtual meetings that last two hours, with the final in-person meeting lasting four 
hours. OCCRL will consult with ICCB and the advisory board on any revisions to the current program review 
manual and on designing the program review process and associated materials for the next full cycle of reviews 
starting in 2022. 


Online Learning Community 


OCCRL will build an online learning community for community college practitioners statewide to support 
effective, efficient, and equity-minded program review and improvement. This community will feature a series 
of forums that allow members of the community, including OCCRL and ICCB, to share knowledge within the 
community and provide tools to support proactive and authentic program assessment and improvements. This 
work will include developing a professional development video series (three to four videos at 20 to 30 minutes 
each) that can be accessed via the learning community as needed during the program review process. Other 
useful information and products will include briefs and podcasts that help support equity-focused, data-driven 
program review practices. 
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Appendix A 


February Focus Group Agenda 


Program Review Illinois Focus Groups 
Radisson Hotel Normal, 8 Traders Circle, Normal, Ill. 61761 


February 8, 2018 
Time Activity Location 
Welcome 
10:00 AM Providing Context: ICCB Goals and Roles in Program White Oak 
Review 
Walkthrough of the Day’s Activities 
10:30 AM_ | Leadership 
Group A: White Oak 
Group B: Hickory Grove 
Group C: Evergreen 
11:15 AM_ | Statewide Guidance and Governance 
11:45 AM__| Lunch, Logistics, and Announcements Evergreen 
12:30 PM _ | Equity-Guided 
Group D: White Oak 
Group E: Hickory Grove 
1:15PM _ | Engagement and Commitment Groep i Eyer 
1:45PM _ | Break and Snack Evergreen 
2:00 PM _ | Data Literacy 
Group G: White Oak 
Group H: Hickory Grove 
Group I: Evergreen 
2:30PM __ | Infrastructure and Information Technologies 
3:00 PM_ | Adjourn 
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Appendix B 
April Focus Group Agenda 


Program Review Illinois Focus Groups 
Radisson Hotel Normal, 8 Traders Circle, Normal, Ill. 61761 


April 5, 2018 
Time Activity Location 
Welcome 
10:00 AM_ | Providing Context: Evidence-Use Cycle White Oak 


Walkthrough of the Day’s Activities 


10:30 AM_ | Producing Evidence 

Group A: White Oak 
Group B: Hickory Grove 
Group C: Evergreen 


11:15AM_ | Sense Making 


12:00 PM_ | Lunch, Logistics, and Announcements White Oak 


12:45 PM_ | Evidence Use 
Group D: White Oak 


Group E: Hickory Grove 
Group F: Evergreen 


1:45 PM_ | Break and Snack White Oak 


Group G: White Oak 
2:00 PM_ ‘| Culture of Evidence Group H: Hickory Grove 
Group I: Evergreen 


3:00 PM | Adjourn 
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Appendix C 


June Focus Group Agenda 


Program Review Illinois Focus Groups 
Radisson Hotel Normal, 8 Traders Circle, Normal, Ill. 61761 


June 7, 2018 


Time Activity Location 
Welcome ; 
TOON Walkthrough of the Day’s Activities Winte ak 
Design Challenges Lie Tega 
10:15 AM e Teams meet 10:15 — 11:10 AM 3 
Teams 5 — 8: Hickory 
e = Report out at 11:15 AM 
Grove 
11:45 AM Lunch, Logistics, and Announcements White Oak 
Review of the CTE Program Review Template aoe Berea Wes 
12:15 PM e Teams meet 12:15 — 1:05 PM F 
Teams 5 — 8: Hickory 
e =Report out at 1:10 PM 
Grove 
1:30 PM Break and Snack White Oak 
Advancing Program Review : : 
1:45 PM © Teams meet 1:45 — 2:25 PM ee pea Ne 
e Report out at 2:30 PM 
3:00 PM Adjourn 
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Appendix D 


Program Review and Change 


Change Made 


How it Supports Students 


Student Populations 
Supported by the Change 


Recruitment of Hispanic 
students 


Increases access for minority students 
More reflective of working population 


Hispanic students 


CTE retention specialist to 
address Perkins metric 


Provides intrusive support to students 
in technology programs 
Meets needs of a special population 


group 


Female students 


Tutoring services moved from 
student services to academic 
services 


All students 


Created a MTH 095 
developmental course for non- 
STEM students 


Attempts to accelerate developmental 
education completion 


All developmental education 
students 


Require a mandatory second 
appointment with advisers 


Provides students additional access to 
advisement and engagement in their 
educational pathways 


All current and new students 


Added additional evening and 
early-morning class session 


Provides added scheduling flexibility 


Working and nontraditional 
students 


Introduction of curriculum 
maps to the CTE and transfer 
faculty 


Maps ensure courses cover all the 
content required for a program 


All students 


Developed a course for faculty 
to learn how to improve 
programs 


Provides students with better 
pathways to certificates and degrees 
Improves course enrollments to 
reduce course cancellations 


Students majoring in low- 
enrollment programs 


Student recruitment for CCA 


Sets limitation on how “low” to dip 
when students enroll in CCA 


Developmental education 
students 


Lab fees changed to facilitate 
timely receipt of necessary 
course materials 


Provides better access to medical 
programs 


Financial aid students 


Focused recruitment efforts for 
nontraditional students 


Provides greater access to programs 


Nontraditional students 


Expansion of mentoring 
programs 


Provides connection and in-class 
support 


All students, students of color, 
developmental education 
students 


Focus on noncognitive and soft 
skill development 


Develops students holistically 


All students 
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